Learning to Gesticulate: Applying Appropriate Animations to Spoken Text

We propose a machine-learning system that learns to choose amongst human-like gestures to accompany novel text. The system is trained on scripts (which are comprised of speech and animations) that were hand-coded by professional animators and shipped in games built on top of the Source game engine. The system first extracts features from the text that was spoken, and maps these features to the gestures that accompany the speech. We have experimented with using a number of features of the text, including n-grams of the words themselves, emotional valence of the speech, and part-of-speech tagging. Using naïve Bayes classifiers, the system learns to associate these features with appropriate gestures. Once trained, our system can be given novel text to which it will attempt to assign appropriate gestures. We examine the accuracy of the system by using n-fold crossvalidation techniques over our training data, as well as a user study, composed of subjective evaluation of the results. In the user study in particular, our system was able to outperform random application of gestures. Although there are many possible applications of automated gesture assignment, in particular we hope to apply this technique to the problem of coordinating human-like gestures to the text spoken by avatars in an automated news show.